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Abstract: In the last years the quality of Higher Education (HE) system and its evaluation have 
been key issues of the political and scientific debate on education policies all over Europe. In 
the wide landscape that involves the entire HE system we draw attention on the third level of 
its organization, i.e. the Ph.D. In particular, this paper discusses the necessity of monitoring the 
recruitment process of Ph.D. system because it represents a fundamental aspect of the Ph.D. 
system as a whole. We introduce a set of concepts related to the recruitment process and then 
we make them computable with synthetic indicators. The study provides an empirical analysis 
based on doctoral schools of four academic years at the University of Siena. Proposed 
indicators are finally used for detecting weakness and strength of each Ph.D. school. 
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1. Introduction 

In the last years , the Doctorates of "Philosophy" (Ph.D.) has gained greater 
importance on the European Higher Education (HE) agenda (Lisbon Agenda, 2000) because 
of the role of doctoral research in contributing to the growth of knowledge and innovation 
(Sursock and Smidt, 2010). In Italy, despite the great relevance given to doctoral studies in 
Europe recently, the availability and transparency of information and indicators for 
monitoring the Ph.D. education system as a whole is still poor and heterogeneous across 
universities (Ghellini et al. 2009). This great heterogeneity in the monitoring of such an 
important stage of HE results in the inability to have an objective view on the state of 
doctoral research in Italy (D'Agostino et al., 2010). In our opinion, also at official level, the 
central institution (MIUR) should require to each university more detailed information about 
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Ph.D. schools. In fact, the official evaluation process refers mainly to the six 
criteria/indicators defined by CNVSU (D.M. n° 290 10/10/2003), which seem not to be well 
specified for having a complete and homogeneous assessment of the courses. For example, 
the sixth criterion states that each Ph.D. course has to satisfy the course objectives, also in 
relation to professional outlets and to the skills achieved during the Ph.D. training; such 
preposition presumes that each university can be able to collect information about the Ph.D. 
process and outcomes. However criteria/indicators referring to outcomes are often missing. 
In some cases, universities have a local monitoring system on Ph.D. outcomes but often they 
are based on ad hoc procedures and surveys which are non-comparable in terms of aims 
and contents, collection methods, questionnaires and reference times (Campostrini, 2010; 
D'Agostino and Ghellini, 2011; Dequarti, Gerzeli and Giudici, 201 0). 

In this complicated framework, we also believe that special attention have to be 
related to recruitment process of Ph.D.s. because this issue is particularly neglected even if it 
has a determinant role in terms of level of prestige of the Ph.D. school and consequentially 
of the academic institution. 

Indeed, also in the lower level of HE the recruitment process is not properly considered as 
done, for example, in UK or in the United States (Aamodt, and Kyvik, 2005, Frolich and 
Stensaker, 2010). Aiming at the recruitment process, the main matters to be considered are: 
a Ph.D. school with several applicants is attractive, a Ph.D. school that uniformly attracts 
students from a variety of regions has an national appreciation, a Ph.D. school that attracts 
students from foreign countries is internationalized, a Ph.D. school that attracts excellent 
students has an elevated degree of quality and again a Ph.D.'s selection strategy which does 
not penalize "external students" has a prominent component of openness. These are the 
crucial issues on which it is important to reflect in order to find appropriate indicators for 
measuring the Ph.D. recruitment process in Italy. 

Normally the most effective recruitment programmes should be those that attract 
large numbers of applicants because "the more you have to choose from, the better the 
choice" (Herriot, 1989) but recruiting the "right" applicants could also be a higher goal. In 
other words, attracting and encouraging more and more candidates to apply to a Ph.D. 
school could be relatively easy, but the recruitment of the "right persons" takes more effort 
and makes a huge difference among Ph.D. schools because this mean to create a talent pool 
of candidates that enables the selection of the best candidates for the Ph.D. school. 

For the above reasons, in this study we focus on recruitment process of Ph.D.s and 
we analyze it on a broad theoretical and empirical basis trying to draw conclusion for 
improving the current assessment of Ph.D. schools. We develop our analysis in a 
multidimensional context by introducing several concepts that we consider essential to cover 
all aspects of the recruitment process and that we believe crucial in order to discriminate 
among Italian Ph.D. schools. Then we put the concept into figures, defining a set of 
indicators, and finally we define a simple measure for aggregating them in order to calculate 
a composite indicator useful for writing a ranking of Ph.D. schools in terms of the 
recruitment process. Anyway we don't give up the multidimensional context as the message 
that we would like to catch is an overview over all the indicators, which allow us to pick up 
for each Ph.D. school the relative position according to the performances of the single 
indicators, highlighting which are the strength and the weakness in each of them. 

Even if the empirical analysis is based on local data relative to four academic years 
(from 2007/2008 to 2010/2011) of the University of Siena, the discussion can be easily 
generalized to a wider context and the definition of objective measures proposed in this 
paper should be useful for different audiences, ranging from prospective graduate students 
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(rankings of Ph.D. schools can influence individual decisions about where to apply), to 
foundations and government funds, to university administrators identifying strength and 
weakness of their school. 

In particular, the paper is organized as follows. In section 2, we briefly describe the 
different aspects that characterize the recruitment process and we introduce the concepts on 
which we develop our analysis. Then in section 3 we present the methodology used. In 
section 4 we discuss our empirical findings and in section 5 we report some final 
considerations. 

2. Discussing on the recruitment process 

2.1 Recruitment and selection phase 

The recruitment process consists of two phases: the recruitment and the selection. 
The recruitment is the process of identifying and attracting a group of potential candidates 
for a particular Ph.D. school, whereas the selection involves the series of steps by which the 
candidates are screened in order to choose the most suitable students for the Ph.D. training. 

The basic purpose of an efficient recruitment process should be to create a pool of 
talent applicants to enable the selection of the best candidates for the Ph.D. school, by 
attracting more and more students who apply, whereas the basic purpose of selection 
process should be to choose the right candidates for the Ph.D. training. 

Generally the recruitment and selection processes are a key factor for the success of 
any business as they help to find the right staff (lies and Salomon,1995) and therefore a 
fortiori they should be a crucial element in the third level of HE. Despite the great 
importance of the Ph.D.s. recruitment process, the assessment of Ph.D. schools in terms of its 
quality is seriously poor in Italy. 

If the recruitment process should consist of practices and activities carried on by a 
Ph.D. school with the primary purpose of attracting potential Ph.D.s, the monitoring of Ph.D. 
process should control the recruitment and the selection policy, with the aim of ensuring, as 
far as possible, that the best students are recruited on merit and that the recruitment process 
is free from bias and discrimination. From this point of view, the first step to pursue is to 
introduce a set of core concepts that can be easily transformed in synthetic measures and 
then used for monitoring the recruitment process as a whole. 

2.2. Key concepts in recruitment process 

We believe that there are at least six key concepts to discuss in this framework: i) 
general attractiveness, ii) diversity, iii) external attractiveness, iv) quality of foreign 
applicants, v) polarization and vi) excellence. 

The importance of the concept of general attractiveness is closely linked to the credit 
that the Ph.D. school has on the market: if a particular Ph.D. school of a peculiar educational 
field is able to attract many applicants with respect with the numbers of grants offered, it 
probably has a greater recognition than another Ph.D. school with less number of applicants 
in the same educational field. Of course, this is a quantitative way of measuring 
attractiveness that certainly is not the only way to measure this concept. Indeed, the 
attractiveness could be also measured using a qualitative assessment, based on information 
collected from applicants through ad-hoc interviews. 

Generally, the meaning of diversity in HE refers to the broader concept of ethnic 
diversity (American Psychological Association Office of Ethnic Minority Affairs, 2000; Gurin et 
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al., 2002). There could be other interesting approaches to consider the diversity, for 
example, from gender, age, religion viewpoint and moreover, in other frameworks of HE, 
the term diversity is also used with regard to variety among programs or services by 
academic institutions (Rossi, 2009; Teixeira et al., 2011). However, in this framework, we 
adopt the concept of diversity with a restricted meaning: indeed, for us diversity means to 
have heterogeneity among applicants in terms of universities where applicants have taken 
their graduation. 

Nowadays, internationalization is wide recognized as one of the most important 
aspects in HE (IAU, 2003) as it constitutes a desirable form of academic diversity. In a wider 
picture, whatever strategy of a Ph.D. school inclines to increase external students (i.e. 
foreigners or applicants coming from other Italian universities) has a prominent component 
of openness. In this view, we refer to external attractiveness. Substantially, it is necessary to 
qualify diversity by measuring if it is directed towards a "local exclusive" shape or on the 
contrary towards completely internationalized shape. 

Following this perspective, another key issue to face is related to the presence of 
foreign students. That is, it's not altogether true that quantity means quality. Indeed, it is well 
known that the variability in HE around the world is wide; for example, education in 
developing countries cannot be compared with well-organized educational systems like in 
Europe. However it is also true that it is better to rank universities than countries. In fact, in 
the world wide university ranking there are also Chinese and Indonesian universities in the 
top 200 (Universities Web Ranking, http://www.4icu.org/top200). However, whatever 
classification is adopted, the crucial aspect to take into account is the reputation of the 
foreigners' applications. In that way we refer to quality of foreign applicants 2 . 

The other important concept that is necessary to clarify is what we call polarization. 
Medium/high degrees of diversity in a Ph.D. school of a particular local institution can be 
due to an elevated presence of students that apply from the other Italian universities. Instead 
of counting the richness (i.e. how many other Italian universities compete in each Ph.D. 
school of this particular local institution), we believe that it is more important to measure if 
there is a particular base (among other universities) that polarizes. In brief, the higher the 
polarization level of Ph.D. schools, the lower will be their diversification in terms of 
applicants. 

Finally, the recruitment of excellent students who are highly motivated and qualified 
for the Ph.D. training has to be another of the priorities of schools as the quality of students 
that a school is able to attract increases its value. Substantially it would be important to 
measure the excellence by the proportion of applicants graduated with a full mark. 

Unfortunately, this concept can have some problems when it has to be translated in 
an objective indicator due to available data. Indeed, in several academic institutions, and 
also in the University of Siena, foreign applicants report their maximum degree mark in a 
not comparable way with the Italian degree with laude. Moreover it is also well-known that 
in Italy there is a high degree of heterogeneity among different educational fields (for 
example, some of them have a greater propensity to give higher marks than others) and 
controlling for this presumable bias effect needs further information 3 . 
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3. Methodological framework 

The methodological framework is multifaceted because the several concepts 
introduced in previous section have to be translated into synthetic measures. In order to 
simplify matters, we present the methodological framework divided into two basic 
perspectives previously mentioned: the attractiveness and the selection process. In particular, 
we pay more attention on the former as available data do not allow to deeply discuss the 
latter. 

The general evaluation question is: is the Ph.D. school attractive? In order to 
establish if a Ph.D. school can be considered attractive we need to evaluate quantitatively 
and qualitatively the applicants. 

Employing this perspective entails a preliminary examination of two population, on 
which indicators have to be referred: the applicants and the population of maintaining 
applicant status. Practically, we deal with the maintaining applicant status as they are the 
real population attending the selection; to simplify the reading, from now, we use the label 
of "applicants" for the population attending the selection. 

A measure for quantifying attractiveness 

For quantifying applicants, in a given academic year, we wonder: how many students 
compete for a grant? The index that we could compute to answer that question is simply the 
ratio between the number of applicants and the number of available grants. Indeed this 
indicator counts how many applicants we have for each grant; it only takes positive values 
and it does not have a maximum, so we prefer to normalize it as follows: 
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where nA (| is the number of applicants and nG,, is the number of available grants in the i-th 
(i—l...l) Ph.D. school belonging to the j-th () educational field. Now the IA index may 
take values in the range [0,1]: it assumes minimum value equal to 0 for the less attractive 
school (the one presenting the smallest ratio nA ,j ) and it assume the maximum value for the 

nG „ 

best attractive school (the one presenting the largest ratio nA o ). The values within the range 

nG ij 

(0,1) indicates the relative position of the generic Ph.D. school in the ranking, with respect to 
the worst or the best Ph.D. school. 

Measures for qualifying attractiveness 

For qualifying attractiveness several features can be considered. Diversity is the first 
of them. This concept relates to the evaluation question: Do the applicants achieve their 
graduation in different places? 

In order to make this concept computable with a synthetic indicator we aggregate 
the information about the provenance of applicants, by using the categorical variable Y, 
whose categories are: a) University of Siena, b) other Italian university, c) foreign university 4 . 

In social science, an index of diversity also referred to heterogeneity Gini index is a 
commonly used measure to determine variation in categorical data (Leti.1983, pag. 266). As 
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the number of categories increases, the maximum value of such index also increases; 
therefore a relative diversity index is adopted and it is defined as: 


ID V = 




k =i 


K 


K -1 


( 2 ) 


where k=l ...K—<3 is the number of categories of Y, and p iik is the proportion of applicants 
in the k-fh category of Ph.D. school i in educational field /. 

ID indicator takes values in the range [0,1]. Low values imply that the Ph.D. school / 
is more concentrated (the minimum value 0 is reached if and only if applicants are perfectly 
homogeneous, that is all of them belong to the same category of the Y) whereas higher 
values mean that the school is more heterogeneous (ID assumes maximum value equal to 1 
for schools whose applicants are perfectly heterogeneous, that is they are equally distributed 
among the categories of the Y variable). 

External attractiveness is the second feature. In order to introduce the corresponding 
indicator related to this concept we start from the consideration that we could have ID=0 
even if all the applicants belong to the category "foreign university", that is certainly 
unusual, but should be desirable in terms of internationalization of the Ph.D. school /. In this 
perspective, the evaluation question is: which is the degree of unlikeness of the observed 
distribution of applicants from the optimal theoretical distribution for a full 
internationalization of the Ph.D. school /? 

So that we assume that categories of variable Y have an ordinal sense as follows: 
"University of Siena", "other Italian university", "foreign university". The internalization index 
which can be viewed as an unlikeness index (Capursi and Librizzi, 2008) is defined as: 


IE ,= [ 


'K-ltl ,J 


(3) 


where P,, k is the cumulative proportion of applicants in the category k (k=l...K<=3) of 
school i in educational field j. IE index takes values in the range [0,1]. A Ph.D. school "local 
(Siena) exclusive" will have an unlikeness score of 0, the index increases if the degree of 
"openness" increases and for a Ph.D. school completely internationalized we will have an 
unlikeness score of 1. 

Polarization is the third feature. Now, for each Ph.D. school we consider only 


applicants belonging to the category "other Italian university". Therefore let 


Piji 


be the 


proportion of students of the Ph.D. school / in educational field / that apply from the l-th 
(1=1...Lii) other Italian university. Substantially we intend to evaluate if, for a given Ph.D. 
school, there is a preponderance (polarization) of applicants from a particular university over 
the "other Italian universities". In that context the evaluation question is: is there a particular 
Italian university (only among other universities), from which the main part of the applicants 
of a given Ph.D. school applies? 

To capture this concept we define the following indicator: 
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This index takes values in the range [0,1]. A Ph.D. school with the minimum degree 
of polarization assumes an index score of 1 (proportion equally distributed among 
universities), the index decreases with the increasing of polarization and we force the index 
to assume a zero value, for a maximum degree of polarization 5 . 

Finally, quality of foreign applicants is the forth feature. Substantially, we move our 
attention to the subset of applicants belonging to the category "foreign applicants". As we 
explained in previous section, it is important for a Ph.D. school to take into account the 
prestige of the universities from which applicants come from. Of course it is a very difficult 
task, because this implies to have a perfect coding of the variable containing the name of 
the degree university for each applicant. In order to bypass that problem we use as proxy of 
the prestige of the university, a measure of "the prestige" of the country to which the 
university belongs; particularly we refer to the Human Development Index (HDI) 6 . So that we 
introduce a simple ratio that we label as "quality of foreign applicants" index as: 


I HDI 


yiF ■■ 
y 

nF: 


VH 


(5) 


where nF\™ is simply the proportion of applicants coming from foreign universities belonging 


to a country with "very high" level of HDI. Obviously the l_HDI index takes values in the 
range [0,1]. A Ph.D. school will be better than others in terms of "quality of foreign 
applicants" if it assumes values close to 1. 


3.1. Measuring the selection process 

The measurement of the selection process is for sure a very hard task. The selection 
should guarantee to choose the most suitable candidates by using the most appropriate, 
efficient, fair, open and effective methods. 

As a preliminary attempt to propose a measure for the assessment of the selection 
process we decided to compute the indexes JD ij and IE.. already calculated for the 

population of applicants, also for the population of winners. The basic idea is: if we assume 
that diversity is one of the most important aim of the recruitment strategy, we can affirm that 
the optimal sharing of the available grants of each Ph.D. school would be similar to the 
observed distribution of applicants. Obviously, observing the values of the indexes computed 
for the winners, we need to take into account of the values that the corresponding indexes 
assume for the applicants, population that we can consider as our conditioning distribution. 

The development of more suitable indicators for measuring the selection process is 
obviously one of our aims but available data for the empirical analysis allow us to calculate 
only the indicators mentioned above that are strongly correlated with the corresponding 
indicators calculated on the population of the applicants. 

3.2. Definition of the composite indicator 

The several indicators defined in the previous sections make difficult to draw up a 
single ranking of the Ph.D. schools. For this reason, we introduce a composite indicator, on 
which a "crude" ranking of the Ph.D. schools can be written. The literature of composite 


117 






The International Conference 
"Innovation and Society 2011. Statistical Methods 
for the Evaluation of Services (IES 2011)" 


JOURNAL 

OF 

APPLIED 

QUANTITATIVE 

METHODS 


indicators offers several examples of aggregation techniques (Saisana et a/., 2005), anyway, 
the definition of a composite indicator is not straightforward and the methodological 
challenges raise a series of technical issues that, if not considered adequately, can lead to 
composite indicators being misinterpreted (Pele, 2009). At official level, in empirical analysis, 
the most adopted formula is the simplest additive aggregation method that entails the 
simple average of sub-indicators having the same measurement unit, therefore we follow 
this methodology. 

In detail, let l iis (s—1 ...S) be the s-th sub-indicator for the school i in the educational 
field /; we define the composite indicator as: 


( 6 ) 



However, in the empirical analysis, equation (6) does not take into account all the 
indicators previously mentioned but only the first four indicators: /A, ID, IE, IP. The l_HDI 
index is neglected because for some Ph.D. schools the number of foreign applicants is 
actually very low. The other two indicators, based on the population of winners, are 
neglected as they are heavy correlated with the respective indicators calculated on the 
applicants. 

However, we are aware that equation (6) may be subject to criticism for its simplicity, 
but in this framework the definition of the composite indicator is not our aim. Indeed, we use 
the composite indicator in order to support a "rule" by which to classify Ph.D. schools without 
losing sight of the multidimensional nature of phenomenon studied. This last aspect will be 
clarify in the empirical analysis. 

4. Empirical analysis 

4.1. Some general figures 

The University of Siena sets up each year a number of Doctoral Schools belonging to 
the following five educational fields: Experimental Science (A), Biomedical and Medical 
Sciences (B), Literature, History, Philosophy and Arts (C), Economics, Law and Political 
Science (D), Interdisciplinary (Areas C and A). 

The admission requirements for each school are specified in the relevant regulations 
for participation and are described in detail on the university web site 
(http://www.unisi.it/internet/home.html ); in general, almost all the schools select 
candidates considering their curricula and an interview. The empirical analysis refers to four 
academic years from 2007/2008 to 2010/2011 and it is based on administrative data 
(SIDRO archive). The number of Ph.D. schools have not a substantial variation across 
specialties and it remains also stable across academic years (we observe, on average, a total 
of 32 schools each years). 

As general information, we know that the number of Ph.D. schools offered by the 
University of Siena in this period of observation shows a different trend, with respect to the 
National landscape (see the XI report CNVSU, CNVSU, 2011). In fact, while at national level 
between 2007/08 and 2009/2010 is observed a decreasing trend, the number of Ph.D. 
courses offered by the University of Siena slightly increases. Whereas, as regard to the 
percentage of the enrolled units who have achieved their graduation in a foreign university, 
the percentage is significantly higher for the University of Siena, with respect to the national 
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figures. Indeed, these latter show a mild but continuous increasing between the 3.2% 
(2003/04) to 8% (2009/10), while the percentage of the enrolled units who have achieved 
the graduation in a foreign university is around the 12% and even greater (around 16%) in 
2009/10. However, the negative evidence, with respect to the national landscape, is that in 
Siena the general trend is uniform and not increasing. 

A positive mark for the University of Siena is that the group of the applicants who 
have achieved their graduation in a foreign university is mainly composed by students 
coming from countries with a very high or high Human Development Index (see HDI in Table 
1) for all the academic years. 


Table 1 . Number of applicants (n) and % by HDI, a.y. 2007/08-2010/11, University of Siena 


HDI 

2007/08 

Academic years 

2008/09 2009/10 

2010/11 


n 

% 

n 

% 

n 

% 

n 

% 

Very high 

52 

4.28 

41 

3.26 

69 

5.08 

59 

4.96 

High 

51 

4.19 

50 

3.98 

58 

4.27 

40 

3.36 

Medium 

29 

2.38 

39 

3.11 

68 

5.01 

35 

2.94 

Low 

9 

0.74 

11 

0.88 

27 

1.99 

8 

0.67 

Italy 

1075 

88.40 

1115 

88.77 

1136 

83.65 

1047 

88.06 

Total 

1216 

100.00 

1256 

100.00 

1358 

100.00 

1189 

100.00 


Some other interesting evidences are related to the number of advertised grants: the 
local declining trend reflects the one related to the courses supply at national level, with the 
strongest declining in a.y. 2009/10. 

4.2. Assessment of Ph.D. schools in terms of recruitment process 

In order to have an adequate number of observations in each Ph.D. school, 
especially for the population of winners, we pooled data across years. We find that in the 
95% of Ph.D. schools the number of maintaining the applicant status is lower than the 
amount of applicants and this kind of selection causes less diversity because the external 
applications decrease. Moreover, there are no significant differences among educational 
fields (see Table 2). 

As explained in section 3, the single indicators presented refer effectively to the 
population of maintaining applicant status as this group represents the students that really 
compete for the selection. 


Table 2. University of Siena-pooling data: number of 
status, winners, grants by educational field 

applicants, 

maintaining 

applicant 

Educational 

field 7 

Applicants 

Maintaining 
applicant status 

Winners 

Grants 

# Ph.D. 

schools 

A & 1 

1059 

881 

360 

215 

11 

B 

917 

676 

316 

203 

9 

C 

1644 

1229 

275 

152 

7 

D 

1399 

990 

218 

138 

10 

Total 

5019 

3776 

1169 

708 

37 
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Using the concepts and the indicators introduced in the previous sections, we 
characterize Ph.D. schools in terms of scores of the composite indicator and the patterns of 
the single indicators. 

As preliminary analysis we compare the mean value and the Coefficient of Variation 
(c.v.), disaggregated by educational fields of each single indicator (see Table 3). For a 
quantitative assessment of attractiveness the /A index is considered. The index highlights a 
significant difference between the average values obtained by the C and D educational fields 
and the ones obtained by A and B 8 . This heterogeneous behavior among areas justifies the 
large value of the c.v. computed for the whole set of Ph.D. schools. Moreover, it is worth to 
note that the D area, the one presenting the largest average value for /A, is also 
characterized by a very high c.v. value. 

For a qualitative assessment of attractiveness the other indexes reported in Table 3 
are considered. Let us begin with the ID and IE couple. The ID index is in general rather 
high (the average on the whole is 0.70) and it means that there is a significant degree of 
diversity. The educational field presenting the smallest ID index is C, however it is the one 
presenting the largest IS value, so it is the most open to applicants coming from foreign 
universities; on the other side we have the B field which is the most heterogeneous and 
consequentially the less internationalized. 

As regard to the IP index, the A, B and D educational fields present similar values 
while C presents the lower value. If we consider this last result jointly with the over¬ 
mentioned, we have that the C area is, on average: very attractive, less diversified than the 
others but the high variability of ID index causes an external attractiveness level similar to D 
area. Finally it seems to suffer of high polarization. 

Looking at IHDI index, its values are significantly greater for A and C educational 
fields, whereas for B and D areas we find lower levels of l_HDI but a very high variability. 
After all, in order to make some conclusion on the selection process we compare the ID and 
IE index computed on the population of applicants with those calculated on the population of 
winners (i.e. /D_w and IE_w indices). The distribution of the winners, on average, seems to 
reflect the one of the applicants for the A and B educational fields; the larger positive 
difference between the two distributions is recorded for the D area, even if we have a very 
good result for this area as regard to the IE indexes; in fact, on average, the distribution of 
the winners reflects the one of the applicants. For the other educational fields, the difference 
between IE and IE_w is always positive, meaning that on average the selection process 
inclines to reduce diversity towards Siena exclusive shape. 


Table 3. Mean and c.v. of single indicators by educational fields 

Educational field 


A B CD Total 


Indicator 

mean 

C.V. 

mean 

C.V. 

mean 

C.V. 

mean 

C.V. 

mean 

C.V. 

IA 

0.13 

0.93 

0.09 

1.17 

0.31 

0.52 

0.32 

0.90 

0.21 

0.99 

ID 

0.77 

0.28 

0.79 

0.13 

0.49 

0.37 

0.69 

0.16 

0.70 

0.27 

IE 

0.44 

0.24 

0.30 

0.33 

0.53 

0.21 

0.50 

0.35 

0.44 

0.33 

IP 

0.31 

0.41 

0.41 

0.31 

0.20 

0.43 

0.34 

0.35 

0.32 

0.41 

IHDI 

0.38 

0.82 

0.22 

1.48 

0.37 

0.67 

0.32 

1.05 

0.32 

0.93 

ID-w 

0.72 

0.17 

0.75 

0.13 

0.60 

0.40 

0.57 

0.44 

0.66 

0.29 

lE-w 

0.33 

0.39 

0.24 

0.29 

0.48 

0.33 

0.50 

0.44 

0.38 

0.48 


The effective assessment of the recruitment process is presented in Table 4 where 
thirty-one schools are listed 9 by their score on the composite indicator defined in section 3.3. 


120 









The International Conference 
"Innovation and Society 2011. Statistical Methods 
for the Evaluation of Services (IES 2011)" 


JOURNAL 

OF 

APPLIED 

QUANTITATIVE 

METHODS 


We decided not to report the scores of each indicators for each school, because, even if the 
quantitative figure are surely much more informative, it results harder to succeed in having 
an overall view of the performance of a Ph.D. school with respect to the others Therefore we 
prefer to report the performance obtained as regard to each indicator / using the 
classification scheme illustrated in Table 5. 

We begin with general observations that can be seen from a cursory inspection of 
the ranking. At the top, the D and A education fields dominate followed by B area, whereas 
at the bottom we find mainly schools of B and C area and also some schools of A 
educational field. 

Particularly, the two best schools belong to D "Economics, Law and Political Science" 
and the following two to A "Experimental Science". Substantially, in this top position the 
schools of D area are characterized by the following pictures: quite high general 
attractiveness, medium level of diversity, very high degree of external attractiveness and 
poor level of polarization. At the same time they attract foreign students of good quality (see 
I HDI indicator). The first school is also able to correct the distribution of applicants toward 
very high level of diversity and internationalization by the selection process. The two schools 
of A education field show poor performances in terms of attractiveness but take advantage 
from a higher diversity, basically due to a high presence of applicants coming from other 
Italian universities among them we do not discover a polarizing academic institution. 

Anyway in the first eleven positions we also find three schools of B area whose 
performances indicate a general lower capacity to attract, with respect to the D education 
field; they present a positive performance as regard to the diversity index (ID), and a 
negative performance regarding to the external attractiveness. However these schools are 
collocated in the top piece of the table thanks to good performance in the IP indicator. 

In the middle of the ranking we find especially schools in C "Literature, History, 
Philosophy and Arts" educational field. They have on average good performances in terms of 
/A index but a poor level of diversity as they tend to high degree of internationalization. 
Moreover they also have very poor performances in terms of polarization. 


Table 4. Ranking of Ph.D. schools based on the recruitment phase, University of Siena - 


pooling data 



Indicators 



Participants 


Winners 


Phd 

schools 

n* 

n** 

Educational 

field 

/A 

ID 

IE 

IP 

/ HD/ 

ID w 

IE w 

jcomp 

23 

177 

28 

D 

+ 

+ 

+ + 

- 

+ 

+ + 

+ + 

0.53 

9 

314 

38 

D 

+ 

0 

+ + 

- 

0 

0 

+ + 

0.53 

33 

29 

12 

A 

- 

+ + 

+ 

+ + 

- 

0 

0 

0.51 

24 

106 

49 

A 

- 

+ + 

0 

+ + 


+ 

- 

0.50 

8 

63 

29 

D 

- 

+ 

+ 

+ 

- 

+ 

- 

0.49 

28 

135 

33 

A 

+ 

+ + 

0 

- 

0 

+ 

- 

0.46 

32 

40 

21 

B 

- 

+ 

- 

+ + 


+ 

- 

0.46 

4 

45 

37 

B 

- 

+ + 

- 

+ 

0 

+ + 

0 

0.45 

34 

113 

20 

A 

+ + 

0 

0 

- 

+ 

+ 

- 

0.45 

12 

69 

25 

B 

- 

+ 

- 

+ + 

- 

+ 

- 

0.44 

2 

54 

27 

A 

- 

+ 

- 

+ 

+ + 

+ + 

- 

0.43 

19 

182 

36 

B 

+ 

0 

+ 


- 

+ 

- 

0.43 

20 

118 

25 

C 

+ 

0 

0 


0 

+ + 

0 

0.43 

5 

138 

65 

B 

- 

+ + 

- 

0 

- 

+ 

- 

0.41 

14 

114 

50 

A 

- 

+ + 

- 


+ 

+ 

- 

0.40 

7 

196 

52 

D 

0 

0 

+ 


- 


+ 

0.40 

18 

47 

31 

B 

- 

+ 

- 

+ + 


+ 

- 

0.40 

16 

373 

71 

C 

+ 

- 

+ 


0 

0 

0 

0.39 
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At the bottom of the ranking we find a mixture of Ph.D. schools belonging to several 
educational fields, indeed several of them show poor attractiveness and some others poor 
diversity and high polarization. Taking, for example, the two bottom PhD schools, their 
external attractiveness performance is good but they heavy suffer of polarization. 


Table 5. Classification system used for defining single index performance 


Intervals 

Classification 

Index 

Performance 

*>3 

1 

VI 

11 —i 

Very poor performance 

// // 

x-s <1 <x- 0.25s 

Poor performance 

// // 

x — 0.25s < I <x + 0.25s 

On average 

" 0 " 

x + 0.25s < / < x + s 

Good performance 

// _j_ // 

+ 

Al 

HH 

Very good performance 

" + +" 


5. Some final remarks 

The ranking exercise has become more and more common in recent years 
despite the widespread criticism of what factors have to be measured and the way 
measurements are carried out. The main purpose of the use of rankings in higher 
education institutions -university, faculty, department- should be to make complex 
circumstances more transparent, especially for students who have to choose in an 
aware way. We also stress that, in Italy, the evaluation of the performance and the 
following ranking would be very important for the highest educational level, i.e. the 
Ph.D. where central institution should invest money in an efficient way in order to 
contribute to the growth of knowledge and innovation. 

We believe that a proper monitoring and evaluation process should start from the 
recruitment phase as it represents the input for the overall Ph.D. process, however this 
aspect is always neglected in the official report that universities have to present at the end of 
each academic year. 

In this view, we propose a final ranking of Ph.D. schools using several aspects of 
Ph.D. recruitment process. Indeed, the final important result of this paper is not the final 
ranking, but it is the possibility of highlighting the strength and the weakness in each Ph.D. 
school for aspects concerning the recruitment process. This information, if properly used, 
with other information can improve the assessment of each Ph.D. school and in the long 
term can help to operate a right rationalization of resources by improving the existing Ph.D. 
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schools in their deficiencies and by decreasing the number of those having several 
shortages. 

As final remark, we add that this paper is intended, not simply to report on the 
current picture, but also to discuss an agenda for presenting actions and future changes. The 
present, in this case could be our enemy. The debate on the present could be our challenge. 
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reported; therefore the country of origin has to be taken into account necessarily. 

3 In our empirical analysis we do not consider this aspect as we check several deficiencies in data. 
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but empirical data do not have this information. 
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